Staccato: Segmentation Agreement Calculator

No matter what your favorite definition of co-verbal “hand and arm gesture” is, at the basic kinematic level a hand and arm gesture is in any case manifested by some spatiotemporal body movement. Accordingly, a prerequisite of any empirical, data-driven account to gesture is to locate those segments in the stream of hand and arm movements that make up a co-verbal hand and arm gesture. Following [3], a gesture basically has a tripartite movement profile, that consists of a preparation, a stroke, and a retraction phase. In addition to these three basic phases, temporary cessations of motion might occur either before or after the stroke might occur in terms of preand post-stroke holds [5]. In the annotation of video data we have to deal with the segmentation problem, that is, the demarcation of the temporal arm and/or hand movements that constitute gestures. Like all annotations, gesture segmentation has to be evaluated with regard to its reliability (see [1] for an introduction into this topic). The standard method for gauging reliability of annotations are chance-corrected assessments of the agreement between multiple annotations of the same material – see [6] for the set-up of a careful evaluation of detailed gesture annotations. However, the reliability of gesture segmentation cannot be assessed by these methods. For standard agreement measures – like the wide-spread kappa statistic [2] – are applicable only to annotation data gained in the a test design that consists in classifying a fixed set of given items (gestures, in our case) into a predefined response categories. The demarcation of movement segments, by contrast, first of all determines what the items are (which then have to be classified according to response categories). Segmentation, therefore, is a precondition for item-based annotations. But how, then, is the reliability of gesture segmentation to be evaluated? Procedures proposed in this regard try to measure the degree of agreement between segmentations of different annotators in terms of some metric gauges. As a metric reference value, the time line or the number of frames of the video film containing the recorded data has been used – cf. the proposals of [4] and, at least partly, [7]. Such frequentist analyzes, however, fail to capture higher order structures like the number or the allocation of marked segments. The core of the assessment problem becomes obvious by the following examples, illustrated in subfigures (a) and (b) of Figure 1. In each of the subfigures segmentations of two annotators are displayed as horizontal lines, indicating the length of the respective segmentation according to the temporal x-axis. In subfigure(a) the annotators agree on the occurrence of a single gesture but merely assign it a different length. In subfigure(b) the annotators identify a different number of gestures. Both cases exemplify two poles of agreement that each pertinent reliability assessment has to kept separate an to account for: 1. annotators in example (a) share a reasonably common view of how the observed gestures have to be segmented; 50